4.4. Evaluations

What is different about an agent evaluation?

An agent answer is the product of multiple model and tool steps. Evaluating only final prose can miss a dangerous or wasteful trajectory. The course stores the prompt, expected tool names/arguments/order, turn boundaries, and a reference response over fixed seed data.

What does a trajectory case look like?

{
  "eval_id": "inventory-status",
  "conversation": [
    {
      "user_content": {
        "role": "user",
        "parts": [{ "text": "What is the status of the inventory service?" }]
      },
      "intermediate_data": {
        "tool_uses": [
          { "name": "get_service_status", "args": { "name": "inventory" } }
        ]
      }
    }
  ]
}

The complete eval set lives in ops.evalset.json — thirteen cases over the fixed seed, containing no real operational data.

How is ADK evaluation gated?

{
  "criteria": {
    "tool_trajectory_avg_score": {
      "threshold": 1.0,
      "match_type": "IN_ORDER"
    }
  }
}

cd agents/python
mise run eval

This is a live-model gate. IN_ORDER requires every expected tool call to appear with the expected arguments, in order, while tolerating extra calls between them — a model may harmlessly recall memory or list incidents first without failing the contract. Tighten to exact matching only when you can explain why any extra call is a defect.

Why do negative and adversarial cases belong in the eval set?

An all-happy-path set measures whether the agent can succeed, never whether it fails safely — it cannot catch a regression that picks the wrong tool for an unknown entity, skips approval, or obeys an instruction planted in tool output. The thirteen cases therefore mix both kinds:

Two design rules keep these cases meaningful:

1. Assert on the tool trajectory, not only the final text. Wording varies between runs and models, but "called get_incident with INC-999 and no other action" is a stable contract — it is exactly what IN_ORDER matching scores.
2. Keep the set consistent with the seed data. The offline suite (test_evalset.py) verifies that every referenced incident, service, and runbook exists in the seed — and that the deliberate negatives INC-999 and warehouse stay missing. When the dataset evolves, dangling references fail mise run test before any model-backed run wastes a comparison.

Grow the set from real failures: when a trace shows a wrong trajectory or an unsafe proposal, distill it into one case that tests that one behavior.

What does the MLflow evaluation add?

mlflow_eval.py converts every conversation turn, executes all turns in one isolated session/state directory, and retains one response and tool trajectory per turn. Four deterministic scorers always run:

@scorer
def tool_trajectory(outputs, expectations) -> bool:
    """Require the expected tool calls per turn, in order (extra calls allowed)."""


@scorer
def complete_conversation(outputs, expectations) -> bool:
    """Require one non-empty terminal response for every expected turn."""


@scorer
def response_facts(outputs, expectations) -> bool:
    """Require stable domain and policy facts from each reference response."""


@scorer
def tool_policy(outputs, expectations) -> bool:
    """Require exact write calls per turn while allowing additional read calls."""

tool_trajectory reimplements the same IN_ORDER semantics as the ADK gate through a shared _in_order helper. Extra reads remain allowed, but tool_policy compares every state-changing call by exact name, arguments, order, and count; an additional/repeated restart, resolution, or memory note fails even if the expected subsequence appeared. response_facts checks stable incident/service/policy terms without forcing exact prose.

A guarded write legitimately stops at ADK's adk_request_confirmation boundary with no assistant text. When that is the terminal event, the evaluator reads only its recognized originalFunctionCall and records a deterministic "waiting for approval; no state change" response so completeness measures a real input-required outcome instead of failing on an empty string. It never sends a confirmation response or runs another model turn; ordinary assistant text wins when present. A real InMemoryRunner regression proves one model call, the restart-plus-confirmation trajectory, and unchanged service state.

The implementation also registers INSTRUCTION, initializes a logged agent model with the prompt URI/version and configured model name, links the evaluation to that model id, and requires all four deterministic metric means to equal 1.0. A missing or lower metric marks the logged model FAILED and exits non-zero instead of logging a false-green result.

Set MLFLOW_EXPERIMENT_NAME to override the default ops-copilot experiment. The command always prints its tracking URI; it emits a local UI hint only when that URI uses SQLite, never when results were sent to an HTTP tracking server.

How does the optional judge work?

Set an explicit model and point the OSS OpenAI client at agentgateway:

MLFLOW_JUDGE_MODEL=qwen3:4b
MLFLOW_JUDGE_BASE_URL=http://127.0.0.1:4000/v1
MLFLOW_JUDGE_API_KEY=local-ollama
mise run eval:mlflow

The judge receives untrusted JSON containing questions, answers, and references, and must return a validated JudgeVerdict. It is optional evidence, not ground truth. The evaluation path does not use LiteLLM or a hidden hosted MLflow service.

All three MLFLOW_JUDGE_* variables are required together. They never fall back to the agent's generic OPENAI_* variables, so enabling a judge cannot silently bypass the intended agentgateway route.

How do you avoid evaluation leakage?

• Keep evaluation cases out of the runtime instruction and retrieved knowledge.
• Use a separate holdout set for release decisions as the corpus grows.
• Do not tune repeatedly against the same thirteen cases and call the result generalization.
• Version data, prompt, tool schema, model, and scorer together.
• Inspect failing traces instead of optimizing only an aggregate score.

Which evaluations run in CI and which stay scheduled?

The merge gate must be fast, deterministic, and secret-free, so .github/workflows/ci.yml runs only offline suites on every push and pull request — including two named steps so a regression is a distinct signal, not a line lost in the full run:

1. mise run redteam — the deterministic adversarial regression suite (see 4.6. Security).
2. mise run eval:validate — evalset consistency: every case references incidents, services, and runbooks that exist in the committed seed, and the trajectory criteria stay strict.

Model-backed evaluation is separate evidence, not a merge blocker. .github/workflows/eval.yml runs weekly and on manual dispatch, only on the canonical repository:

1. It provisions a local Ollama server on the runner and pulls a small open model (default qwen3:4b; dispatch a smaller tag to trade quality for speed on the 4-vCPU runner).
2. It runs mise run eval, mise run eval:report, and mise run eval:mlflow against the fixed seed data, without the optional judge, and uploads both model-backed logs, the SQLite tracking store, and the MLflow run artifacts as a downloadable workflow artifact.
3. It needs no provider secret, so forks and pull requests never require credentials — and it never triggers on pull requests.

The split is deliberate: trajectory scores move when the model, prompt, or tools change, so they belong on a schedule where a failure means "inspect the results", while the merge gate only fails on regressions a developer can fix deterministically.

What is the evaluation checkpoint?

Run mise run test first. With an explicitly configured model, run mise run eval and mise run eval:mlflow, then record the model name, prompt URI/version, dataset commit, scores, and failing cases. Do not make live-model calls merely to satisfy the offline chapter gate.

How would you add an evaluation case?

Exercise: extend the eval set with a case that pins a behavior you care about.

• Goal: add one new case to the eval set (a negative or adversarial one is most valuable) that asserts a specific behavior — a refusal, a required tool call, or a schema-shaped answer.
• Files to touch: agents/python/evals/ops.evalset.json (or triage-report.evalset.json), and confirm it is exercised by the deterministic validation in agents/python/tests/test_evalset.py.
• Gate that proves completion: cd agents/python && mise run eval:validate passes (the new case validates against the seed data), and a model-backed mise run eval scores it as expected.